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FOREWORD 


The  work  described  in  this  report  was  initiated  under  program  element  63720N, 
project  Z1175-PN  (Training  System  Design  and  Management),  subproject  05  (Improved 
Effectiveness  in  Course  Design,  Delivery,  and  Evaluation)  and  was  sponsored  by  the 
Deputy  Chief  of  Naval  Operations  (Manpower,  Personnel,  and  Training)  (OP-01).  Addi¬ 
tional  funding  was  provided  by  the  Chief  of  Naval  Education  and  Training. 

^The  objective  of  the  subproject  was  to  develop  an  empirically-based  instructional 
design  support  system  to  aid  developers  in  choosing  instructional  alternatives  based  on 
cost/benefits  and  specific  resource  limitations.  The  objective  of  this  report,  which  is 
intended  for  course  designers  and  developers  and  those  managing  instructional  develop¬ 
ment,  was  to  describe  the  development  and  evaluation  of  the  instructional  quality 
inventory  (IQI),  a  systematic  methodology  for  reviewing  instructional  material.  The 
intent  is  to  improve  the  quality  of  the  materials,  thereby  increasing  the  effectiveness  of 
later  student  tryouts. 

\ 

The  work  on  automating  the  IQI  is  being  funded  under  subproject  Z1388-PN.01  (Low- 
cost  Microcomputer  Training  Systems). 


1.  W.  RENARD  3AMES  W.  TWEEDDALE 

Commanding  Officer  Technical  Director 
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SUMMARY 


Problem 


The  Navy  has  adopted  the  Instructional  Systems  Design  (ISD)  technology  for  the 
development  of  its  training  programs.  Although  there  is  a  general  concern  for  the  quality 
of  the  products  developed  using  this  technology,  there  are  no  techniques  for  evaluating 
these  products  prior  to  conducting  student  tryouts.  Because  of  the  expense  of  making 
extensive  revisions  following  student  tryouts,  the  Chief  of  Naval  Education  and  Training 
(NAVEDTRA  110A,  1981)  has  directed  the  designer  and  a  subject  matter  expert  to  perform 
an  "internal  review"  of  preliminary  instruction  for  judging  accuracy  and  completeness,  the 
adequacy  of  the  instructional  sequence,  the  motivating  capability,  and  the  support  for 
student  learning.  However,  no  method  is  specified  for  performing  this  review. 

Objective 

The  objective  of  this  effort  was  to  review  the  development  and  evaluation  of  the 
instructional  quality  inventory  (IQI),  a  systematic  methodology  for  reviewing  the  three 
major  products  of  the  ISD  process-objectives,  test  items,  and  instruction— before 
conducting  student  tryouts. 

IQI  Development  and  Evaluation 

The  IQI  was  developed  to  assist  in  the  formative  evaluation  of  instruction.  It  was 
tested  empirically  in  experiments  that  examined  the  validity  of  its  individual  prescriptions 
and  in  studies  that  demonstrated  its  applicability  and  utility.  In  addition,  it  underwent 
extensive  formative  evaluation  in  a  series  of  over  50  workshops. 

Conclusions 


The  IQI  is  an  effective  empirically-based  methodology  for  reviewing  objectives,  test 
items,  and  instruction  for  consistency  ind  adequacy.  It  can  be  used  by  instructional 
development  personnel  after  a  minimal  amount  of  training.  However,  it  is  not  designed  to 
assist  developers  in  adapting  instructional  programs  to  individual  student  learning  styles, 
experiential  background,  or  preferences.  Further,  it  does  not  address  the  overall 
structure  of  the  instructional  content  and  how  best  to  sequence  the  instructional  program. 

Recommendations 


1.  It  is  recommended  that  instructional  developers  use  IQI  to  (a)  assist  in  the 
format! /e  evaluation  of  newly-developed  instruction,  (b)  review  existing  instruction  for 
consistency  and  adequacy,  and  (c)  provide  standards  for  contractor-developed  instruction. 

2.  It  is  recommended  that  IQI  procedures  be  automated  to  (a)  minimize  skill 
deficiencies  in  instructional  development  personnel,  (b)  allow  designers  to  pay  more 
attention  to  the  difficult  tasks  of  analysis  and  design,  and  (c)  speed  the  internal  review 
process  and  facilitate  revisions. 
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INTRODUCTION 


Problem 


The  Navy  has  adopted  the  Instructional  Systems  Design  (ISD)  technology  for  the 
development  of  its  training  programs  (Branson,  Rayner,  Cox,  Furman,  King,  &  Harnum, 
1975;  Branson,  1977,  1979).  Although  there  is  a  general  concern  for  the  quality  of  the 
products  developed  using  this  technology,  there  are  no  techniques  for  evaluating  these 
products  before  conducting  student  tryouts.  Because  of  the  expense  of  making  extensive 
revisions  following  student  tryouts,  the  Chief  of  Naval  Education  and  Training  (NAVED- 
TRA  110A,  1981)  has  directed  the  designer  and  a  subject  matter  expert  (SME)  to  perform 
an  "internal  review"  of  preliminary  instruction  for  judging  accuracy  and  completeness,  the 
adequacy  of  the  instructional  sequence,  the  motivating  capability,  and  the  support  for 
student  learning.  However,  no  method  is  specified  for  performing  the  review. 

Objective 

The  objective  of  this  report  is  to  describe  the  development  and  evaluation  of  the 
instructional  quality  inventory  (IQI),  a  systematic  methodology  for  reviewing  the  three 
major  products  of  the  ISD  process— objectives,  test  items,  and  instruction— before 
conducting  student  tryouts. 


IQI  DEVELOPMENT  AND  EVALUATION 


Background 

Current  approaches  to  the  management  of  instructional  design  and  development 
combine  methods  for  systems  analysis  or  operations  research  with  learning  and  instruc¬ 
tional  psychology.  The  armed  services  have  all  adopted  the  ISD  methodology  for 
accomplishing  instructional  design  and  development  (Branson  et  al.,  1975;  Branson,  1977, 
1979).  Although  there  are  weaknesses  in  the-  proceduraiized  guidebooks  that  support  ISD 
and  its  management  (Montague  <5c  Wulfeck,  1982;  Vineberg  &  Joyner,  1980;  Wetzel,  Ellis, 
Wulfeck,  <5c  Montague,  1982),  they  do  provide  basic  guidance  for  analyzing,  designing, 
developing,  evaluating,  and  imp.ementing  instruction.  Since  all  instruction,  whether 
developed  formally  or  informally,  goes  through  those  stages  or  phases,  it  is  generally 
agreed  that  this  focus  on  task-referenced  instruction  and  the  exploitation  of  modern 
communications  technology  for  instruction  is  a  desirable  goal  (e.g.,  Dick  <5c  Carey,  1978; 
Gropper,  1980;  Gagne  in  Briggs,  1979,  O'Neil,  1979a,  1979b). 

A  major  problem  with  ISD  is  that  it  depends  on  the  designer/developer's  expertise. 
How  well  the  stages  are  carried  out  depends,  in  large  part,  on  the  organization,  artistry, 
and  effort  put  into  them.  Sometimes  various  steps  are  left  out  or  ignored,  "new" 
instruction  adopts  old  materials  by  default,  tests  are  inconsistent  with  instructional 
requirements,  materials  are  inadequate,  etc.  In  general,  ISD  is  long  on  "what  to  do"  but 
short  on  "how  to  do  it."  For  example,  at  several  points  in  the  design  sequence, 
intermediate  products  are  developed,  such  as  objectives,  test  items,  and  segments  of 
instruction  to  support  learning.  Although  there  is  a  general  concern  about  the  quality  of 
the  products  being  developed,  no  techniques  are  given  for  evaluating  them  before  student 
tryouts  are  conducted. 

In  fact,  major  guides  for  design  and  development  describe  formative  evaluation 
almost  exclusively  in  terms  of  such  tryouts  (e.g.,  Dick,  1977;  Gagne<5c  Briggs,  1979;  Bloom, 
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Hastings,  <Sc  Madaus,  1971;  Tuckman,  1979).  Since  the  design  process  is  based  on 
approximate  and  imperfect  knowledge,  such  tryouts  are  vital  to  detect  inadequacies  in  the 
materials.  For  example,  in  his  widely  cited  chapter,  Dick  (1977)  describes  three  stages  or 
phases  of  formative  evaluation.  First,  lesson  materials  are  tried  with  students  one-at-a- 
time,  so  that  glaring  faults  and  misunderstandings  can  be  corrected.  Second,  small  group 
trials  are  held,  and  further  revisions  are  made  as  necessary.  Third,  the  prototype  is  tested 
in  the  standard  environment  under  the  appropriate  operating  conditions.  Final  revisions 
are  based  on  various  forms  of  information,  including  test  performance,  student  and 
instructor  questions  and  comments,  etc.  Because  of  the  expense  of  this  process,  internal 
review  has  recently  received  attention. 

Merrill  and  his  associates  (e.g.,  Merrill  <Sc  Boutwell,  1973;  Merrill,  Richards,  Schmidt, 
<5c  Wood,  1977),  who  were  working  on  the  problem  of  standards  for  the  instructional 
development  process,  were  initially  concerned  with  deriving  instructional  prescriptions 
from  research  studies  on  instructional  variables.  They  suggested  that  such  prescriptions 
served  not  only  to  recommend  instructional  practices  but  also  as  a  basis  for  reviewing 
existing  instructions  for  internal  consistency  and  adequacy.  Subsequently,  they  developed 
the  instructional  strategy  diagnostic  profile  (ISDP)  (Merrill  et  al.,  1977).  The  ISDP 
procedures  underwent  extensive  field  testing  and  revision  at  the  Navy  Personnel  Research 
and  Development  Center,  and  resulted  in  the  IQI.  Merrill,  Reigeluth,  and  Faust  (1979) 
described  a  slightly  different  version  of  the  procedures,  and  Reigeluth  (1980)  suggested 
using  them  to  train  teachers  to  Improve  classroom  teaching. 

IQI  Classification  Scheme 


IQI  was  designed  initially  to  parallel  and  supplement  the  military's  ISD  model. 
However,  it  can  be  applied  to  any  systematically  developed  program  of  instruction  that 
has  objectives,  test  items,  and  instruction  tied  to  the  objectives,  since  it  focuses  on  those 
three  main  parts.  It  is  intended  for  use  with  cognitive  and  psychomotor  instruction  and  is 
appropriate  for  evaluating  most  forms  of  such  materials  (cf.,  Roid  &  Haladyna,  1982). 

IQI  uses  a  scheme  that  classifies  objectives,  test  items,  and  instructional  presenta¬ 
tions.  Classification  is  useful  for  several  reasons: 

1.  It  helps  make  more  precise  judgments  about  the  adequacy  of  learning  objectives 
and  leads  to  more  precise  test  item  specifications. 

2.  It  assists  the  internal  review  process  by  guiding  judgments  about  the  consistency 
between  objectives  and  test  items.  If  objectives  and  test  items  were  not  classified,  all 
one  could  say  is  "This  is  an  objective  and  this  is  a  test  item,  and  they  don’t  look  too 
different." 

3.  It  not  only  ensures  that  objectives,  test  items,  and  the  corresponding  instruction 
address  the  same  learning  and  performance  requirements,  but  also  helps  to  judge  whether 
or  not  they  are  adequate. 

Classification  schemes  have  been  used  in  the  past  with  some  success.  However,  if  a 
classification  scheme  is  to  be  useful,  it  must  meet  two  conditions.  First,  instructional  and 
test  developers  must  be  able  to  make  reliable  classifications  using  the  scheme.  Second, 
the  scheme  must  have  clear  implications  for  instructional  and  test  item  development;  the 
implications  should  be  specified  as,  prescriptions  for  development.  Typically,  classifica¬ 
tion  procedures  are  too  loosely  defined  (e.g.,  Gagne  ,  1976;  Popham,  1978)  to  permit 
reliable  classification  by  users  who  have  not  had  considerable  training  or  who  are  not 
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highly  sophisticated  instructional  designers.  Although  military  instructor  personnel  have 
difficulty  in  using  schemes  of  this  nature,  they  do  the  majority  of  instructional  and  test 
development.  In  addition,  the  implications  of  classification  for  instruction  and  testing  are 
often  not  clearly  specified  (Briggs,  1977).  The  1QI  classification  scheme  has  been  tested 
and  extensively  revised  (Ellis  <Jc  Wulfeck,  1979)  and  is  currently  in  wide  use  by  Navy 
personnel.  In  its  present  form,  it  meets  the  two  conditions  listed  above  (Ellis  &  Wulfeck, 
1982). 

Objectives  and  test  items  can  be  classified  according  to:  (1)  what  the  student  must 
do  (i.e.,  the  task  to  be  performed),  and  (2)  the  instructional  content  (i.e.,  the  type  of 
information  the  student  must  learn).  The  IQl  task/ content  matrix  is  displayed  in  Table  1 
and  discussed  below: 

1.  The  Task  Dimension.  A  student  can  either  remember  information,  or  use  the 
information  to  do  something.  This  distinction  corresponds  to  the  difference  between 
knowledge  and  application  and  to  the  difference  between  declarative  and  procedural 
knowledge  (Ryle,  1949;  Broudy,  1977).  The  use  level  can  be  further  divided  into  two  types: 
(a)  use-unaided,  where  the  person  has  no  aids  except  his  or  her  own  memory,  and  (b)  use- 
aided,  where  the  person  has  a  job  aid  designed  to  reduce  dependence  on  memory. 

2.  The  Content  Dimension.  There  are  five  content  types: 

a.  Facts,  which  are  simple  associations  between  names,  objects,  symbols, 
locations,  etc.,  can  only  be  remembered  while  the  other  content  types  can  be  remembered 
or  used. 


b.  Categories  are  classifications  defined  by  certain  specified  characteristics. 

c.  Procedures  consist  of  ordered  sequences  of  steps  or  operations  performed  on 
a  single  object  or  in  a  specific  situation. 

d.  Rules  also  consist  of  ordered  sequences  of  operations  but  can  be  performed 
on  a  variety  of  objects  or  in  a  variety  of  situations. 

e.  Principles  involve  explanations,  predictions,  or  diagnoses  based  on  theoret¬ 
ical  or  cause-effect  relationships. 


Table  1 

TASK-CONTENT  MATRIX 


Task  levels 


Content  Types 


Fact 


Category 


Procedure 


Rule 


Principle 


Remember 


Recall  or 
recognize, 
names,  parts, 
dates,  etc. 


State  the 
basis  for  clas¬ 
sifying  the 
characteris¬ 
tics. 


List  steps 
used. 


State  the 
rule  that 
applies  to 
these  situa¬ 
tions. 


State  the 
principle  that 
explains 
these  events. 


Use-unaided 


Not 

ble. 


applies- 


Classify 
objects, 
events,  by 
characteris¬ 
tics. 


Do  steps  in 
this  context 
or  on  this 
equipment. 


Use  rule  or 
formula  on 
these  types 
of  tasks  or 
new  prob¬ 
lems. 


Explain  or 
predict 
event(s) 
using  princi¬ 
ple. 


Use-aided 


Not 

ble. 


applica- 


Given  steps, 
do  them. 


Given  for¬ 
mula,  use  to 
solve. 


Given  princi¬ 
ple,  use  to 
explain. 


Given 
characteris- 
tics,  classify. 
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In  summary,  the  remember  level  involves  simply  remembering  facts  and  the  steps, 
descriptions,  or  definitions  of  categories,  procedures,  rules,  and  principles.  The  use- 
unaided  level  involves  remembering  what  is  to  be  done  and  then  doing  it.  The  use-aided 
level  involves  doing  tasks  using  a  job  aid. 

1QI  Procedures 

The  IQI  consists  of  five  procedures,  which  are  listed  in  Table  2. 


Table  2 

IQI  EVALUATION  PROCEDURES 

Instructional  component 

Procedures 

Classified  Objectives 

1.  Judge  adequacy  of  objcctivea/go&ls. 

Tests  and  test  items 

2.  Are  objective*  and  test  items  consistent? 

3.  Are  test  items  adequate? 

Instruction/Presentations 

4.  Are  presentations  consistent  with  objectives? 

5.  Are  presentations  adequate? 

Since  all  the  steps  of  the  instructional  development  process  depend  on  careful 
specification  of  learning  objectives,  the  first  procedure  is  to  assure  the  adequacy  of 
objectives.  This  is  done  by  classifying  each  objective,  determining  whether  all  the 
necessary  conditions,  standards,  and  actions  are  present,  and  judging  whether  or  not  the 
objective  accurately  reflects  what  the  student  is  supposed  to  do  or  know  following 
training. 

The  next  two  procedures  involve  checking  test  items.  First,  test  item(s)  are 
evaluated  against  their  corresponding  objectives  to  ensure  that  the  test  items  are 
consistent  with  the  objectives.  This  is  accomplished  by  making  sure  that  the  conditions, 
standards,  actions,  task  level,  and  content  type  for  the  test  items  are  the  same  as  those 
for  the  objective.  Consistency  problems  occur  frequently,  even  in  systematically 
developed  instruction.  It  is  likely  that  restrictions  (e.g.,  in  testing  time,  laboratory  space, 
etc.)  can  cause  instructors  to  develop  inconsistent  tests.  For  example,  Lockhart,  Sturges, 
Van  Matre,  and  Zachai  (1981)  found  that  79  percent  of  the  test  items  in  a  recently 
developed  Navy  technical  course  were  faulty,  and  over  40  percent  of  these  involved 
inconsistencies  between  objectives  and  test  items.  Next,  the  adequacy  of  the  test  items 
is  assessed  by  determining  whether  each  item  conforms  to  criteria  for  proper  item 
construction. 

At  this  point,  the  objectives  and  test  items  for  the  instructional  program  are 
consistent  with  each  other  and  are  adequate.  The  next  step  is  to  ensure  that  the 
instructional  presentation  (e.g.,  printed  self-study  materials,  lectures,  computer-assisted 
instruction,  films,  etc.)  is  consistent  with  the  objectives  and  test  items;  that  is,  it  must 


teach  to  the  task  level  and  content  type  of  the  objective/test  item.  Different 
combinations  of  instructional  components  (e.g.,  expository  statements,  examples, 
practice,  feedback),  are  required,  depending  on  the  task  level  of  the  objective.  A 
component  is  counted  as  present  in  the  instruction  only  if  it  is  complete.  This  depends  on 
the  content  type  of  the  objective  and  simply  means  that  everything  that  needs  to  be 
included  is  included. 

Since  instruction  can  be  consistent  but  still  not  teach  as  effectively  as  it  could,  the 
final  procedure  evaluates  the  adequacy  of  the  instructional  presentation.  The  adequacy 
judgment  incorporates  a  number  of  instructional  design  principles  that  have  been  shown  to 
promote  student  learning,  including  guidelines  for  formatting  information  so  students  can 
find  it,  communicating  it  clearly  and  effectively  so  students  can  understand  it,  and 
providing  additional  explanation,  so  students  can  better  learn  and  retain  the  information. 

The  IQI  consists  of  the  following  four  volumes,  which  are  available  from  the  Defense 
Technical  Information  Center,  Alexandria,  Virginia  (see  references): 

1.  Introduction  and  Overview  (Wulfeck,  Ellis,  Richards,  Wood,  6c  Merrill,  1978). 

2.  Users'  Manual  (Ellis,  Wulfeck,  6c  Fredericks,  1979). 

3.  Training  Workbook  (Fredericks,  1980). 

4.  Job  Performance  Aid  (Ellis  6c  Wulfeck,  1978). 


This  section  reviews  some  of  the  experimental  tests  of  the  IQI  prescriptions,  some 
attempts  to  demonstrate  its  utility  and  validity  in  diagnosing  and  revising  inconsistent  and 
inadequate  instruction,  and  its  own  formative  evaluation. 


Empirical  Studies 


Merrill,  Reigeluth,  and  Faust  (1979),  in  reviewing  existing  pertinent  research,  found 
considerable  support  for  IQI  prescriptions.  Of  51  studies  that  were  valid  tests  of  IQI 
principles,  none  contradicted  any  of  its  principles.  In  addition,  they  conducted  a  number 
of  empirical  studies  to  test  (1)  the  consistency  between  instruction  and  test  items,  (2) 
prescriptions  about  the  separation  of  statements,  examples,  and  practice,  and/or  (3)  the 
effects  of  attention-focusing  helps  for  classification  tasks,  elaborative  he!,s  for  state¬ 
ments  and  examples,  and  instance  divergence  and  matching  for  classification  tasks. 
Results  of  all  studies  supported  IQI  prescriptions  and  principles. 


Merrill,  Wood,  Baker,  Ellis,  and  Wulfeck  (1978)  conducted  a  study  to  empirically 
validate  som-'  of  the  consistency  and  adequacy  prescriptions  of  the  IQI.  Specifically,  the 
study  was  designed  to  test  six  hypotheses  concerning  the  consistency  and  adequacy  of 
instructional  materials.  Subjects  participating  in  the  study  were  U.S.  Navy  enlisted  men 
waiting  to  begin  "A"  School  at  the  Propulsion  Engineering  (PE)  School,  Great  Lakes. 
Instructional  materials  developed  were  based  on  PE  school  curriculum  but  were  adapted  to 
provide  for  12  experimental  treatments  needed  to  test  the  six  hypotheses.  Four  of  these 
treatments  represented  remember-level  instruction;  and  eight,  use-level  instruction. 
After  students  finished  their  instruction,  they  were  tested  on  remember-level  test  items 
(labeling  and  listing)  and  use-level  test  items  (classification).  All  subjects  had  the  same 
test'ng  materials.  Three  experiments  were  conducted,  which  are  described  below. 


1.  Experiment  I  tested  the  consistency  hypothesis  (i.e.,  performance  will  decrease 
if  test  items  and  the  instruction  are  not  consistent)  by  manipulating  test  items  and  the 
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inst;  uction.  Results  showed  that  the  students  in  the  use-level  treatment  groups  scored 
significantly  higher  on  use-level  items  (classification)  than  did  those  in  the  remember- 
level  groups,  and  students  in  the  remember-level  groups  scored  significantly  higher  on 
remember-level  items  (labeling  and  listing)  than  did  those  in  the  use-level  groups.  Thus, 
the  consistency  prescription  of  the  IQI  was  supported. 

2.  Experiment  II  tested  the  adequacy  hypothesis  for  remember-level  items  (i.e., 
performance  will  increase  with  the  use  of  a  mnemonic  or  several-page  distributed 
practice)  by  comparing  the  performance  of  (a)  a  mnemonic  group  and  a  no-mnemonic 
group  and  (b)  a  several-page  distributed-practice  group  and  a  one-page  massed-practice 
group.  Results  showed  that  there  were  no  significant  differences  in  performance  on  the 
remember-level  test  items  for  either  the  mnemonic  or  practice  variables,  but  there  was  a 
significant  savings  of  time  for  the  massed-practice  condition.  Thus,  the  two  presentation 
adequacy  prescriptions  of  the  IQI  for  remember-level  items  were  not  supported. 

3.  Experiment  III  tested  the  adequacy  hypotheses  for  use-level  items  (i.e.,  perfor¬ 
mance  will  increase  with  the  use  of  definitions  that  are  separated  from  the  rest  of  the 
instructional  presentation,  divergent  examples,  and  attribute  isolat.on  elaboration).  This 
was  done  by  comparing  the  performance  of  (a)  a  separated-defnition  group  and  an 
embedded-definition  group,  (b)  a  divergent-example  group  and  a  convergent-example 
group,  and  (c)  an  attribute-isolation  group  and  a  no-attribute-isolation  group.  Results 
showed  that  students  given  separated  definitions  scored  higher  on  all  performance 
measures  and  took  less  time  than  did  those  in  the  embedded-definition  treatment. 
Students  in  the  divergent-example  treatments  scored  higher  on  use  test  items  than  did 
those  in  the  convergent-example  treatment.  There  were  no  differences  between  students 
in  the  attribute-isolation  and  no-attribute-isolation  elaboration  treatments.  Thus,  only 
two  of  the  three  presentation  adequacy  prescriptions  of  the  IQI  for  use-level  items  were 
supported. 

Demonstration  Studies 

Two  studies  were  conducted  to  assess  the  IQI's  usefulness  for  revising  existing 
segments  of  instruction  and  to  compare  the  effectiveness  of  the  original  and  revised 
materials. 

Study  I.  IQI  procedures  were  applied  to  insti  uctional  materials  taken  from  a  Navy 
radioman  course  that  described  several  types  of  Navy  call  signs  (call  signs  are  used  to 
identify  radio  stations  (e.g.,  WABC  or  KNBC)),  and  a  revised  version  was  developed.  The 
IQI  and  school  versions  were  compared  using  two  groups  of  20  Navy  subjects.  The 
objectives  for  the  lesson  required  students  to  (1)  recall  the  names  and  defining  character¬ 
istics  of  five  types  of  Navy  call  signs  (remember-cctegory  task),  and  (2)  to  classify  new 
call  signs  according  to  one  of  the  five  types  (use-category  task).  After  completing  the 
materials,  study  time  was  recorded,  and  subjects  were  asked  to  recall  a  written  list  of  the 
names  and  definitions  of  the  call  signs  and  to  classify  a  list  of  18  call  signs.  Dependent 
measures  included  the  times  required  to  complete  the  materials  and  tests  and  the  number 
corect  on  the  tests. 

Results  showed  that  IQI  subjects  performed  significantly  better  on  both  tests  than  did 
school  subjects.  Means  for  the  IQI  group  for  the  recall  and  classification  tes.s  were  83 
and  81  percent  respectively,  compared  to  49  and  60  percent  for  the  school  group.  There 
were  no  differences  in  time  to  complete  the  materials  but  IQI  subjects  took  less  time  to 
complete  the  tests. 


Study  II.  The  objectives  of  this  research  were  to  (1)  determine  whether  relatively 
inexperienced  persons  could  use  IQI  to  identify  difficulties  in  and  revise  instructional 
materials,  (2)  evaluate  the  effectiveness  of  the  revised  materials,  and  (3)  determine  costs 
ci  applying  the  IQI  (Stern  &  Fredericks,  1982).  A  lesson  for  Navy  radiomen,  which  trains 
students  to  verify  spelling,  punctuation,  and  character  alignment  on  a  special  optically 
scanned  form,  was  selected  since  it  had  been  identified  by  instructors  as  needing  revision. 
A  two-person  team  with  limited  experience  in  the  use  of  the  IQI  process  analyzed  the 
selected  material,  and  examined  the  objectives,  testing  format  and  content,  and  training 
materials.  This  resulted  in  (1)  some  changes  to  the  objectives,  which  were  found  to  be 
inconsistent  with  job  requirements,  (2)  an  entirely  different  approach  to  testing,  and  (3)  a 
new  set  of  training  materials.  During  the  process,  lessonware  development  costs  were 
tracked  to  allow  for  later  cost  analysis. 

To  evaluate  the  effectiveness  of  the  revised  materials,  two  groups  of  30  students 
were  compared  on  learning  from  the  old  and  revised  lesson.  Results  showed  that  the 
groups  differed  significantly  on  their  test  performance.  The  average  score  for  IQI 
students  was  57  percent  correct,  compared  to  49  percent  for  standard  students.  The 
standard  group  had  studied  their  materials  for  an  average  of  11.6  hours,  compared  to  2.8 
hours  for  the  IQI  group.  Thus,  the  IQI  group  scored  somewhat  higher  on  the  job-relevant 
test  than  did  trainees  in  the  standard  group  and  learned  more  efficiently.  In  addition  to 
the  empirical  findings,  the  IQI  provided  a  useful  framework  for  determining  deficiencies 
in  current  instruction.  By  examining  module  components  in  the  sequence  called  for  by 
IQI,  deficiencies  could  be  readily  identified  and  revisions  made.  Finally,  an  important 
consideration  in  revising  instruction  concerns  the  costs  incurred  in  the  process.  These 
were  almost  all  in  hourly  costs  for  the  research  team,  amounting  to  one-fifth  of  a  man- 
year.  However,  it  is  estimated  that  this  figure  would  be  considerably  reduced  if  IQI  were 
applied  concurrently  with  the  1SD  process  and  would  be  less  as  the  team  gained 
experience.  The  findings  suggest  that  the  IQI  material  is  more  effective  in  producing  job¬ 
relevant  performance  than  are  existing  formative  evaluation  methods. 

Formative  Evaluation  Workshops 

In  addition  to  the  empirical  studies,  the  IQI  procedures  were  subjected  to  a  rigorous 
formative  evaluation.  A  series  of  over  50  workshops  involving  several  hundred  potential 
users  were  conducted  (Wood,  Ellis,  <5c  Wulfeck,  1978).  Participants  in  these  workshops 
were  asked  to  comment  on  and  criticize  the  validity  and  utility  of  the  IQI  procedures. 
The  information  gained  from  conducting  the  workshops  was  used  to  revise  and  improve  the 
IQI. 


DISCUSSION  AND  CONCLUSIONS 


Usefulness  and  Validity 

The  main  contribution  of  the  IQI  is  that  it  synthesizes  research  knowledge  on 
instruction  in  a  form  that  is  implementable.  The  underlying  principles  are  not  new.  They 
were  derived  from  the  general  research  literature  and  were  confirmed  by  studies 
specifically  designed  tc  test  its  prescriptions.  Since  no  studies  have  been  found  that 
contradict  the  bases  for  quality  assessment  in  the  IQI,  it  appears  that  its  underlying 
conceptions  are  sound. 

The  demonstration  studies  attempted  to  use  the  IQI  in  operational  environments. 
Study  I  demonstrated  that  experienced  personnel  could  revise  existing  course  materials 
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with  corresponding  significantly  increased  student  test  performance.  Study  II  (Stern  Sc 
Fredericks,  19S2)  demonstrated  the  effectiveness  of  the  efforts  of  less  experienced 
personnel.  The  IQ1  is  an  effective  tool  that  can  be  used  with  a  limited  amount  of  training. 
The  main  reason  for  its  acceptance  appears  to  be  that  it  provides  a  systematic  framework 
for  applying  current  knowledge  about  instructional  strategies  and  for  managing  instruc¬ 
tional  program  development. 

The  IQI  framework  has  also  proven  useful  as  a  basis  for  guiding  the  construction  of 
tests  of  student  learning.  Performance-based  training  requires  the  use  of  tests  that  are 
criterion-referenced;  that  is,  that  directly  measure  aspects  of  performance  required  in 
course  objectives.  In  addition,  during  instruction,  it  is  important  to  be  able  to  determine 
why  a  student  cannot  perform  well  so  that  the  problem  or  errors  can  be  corrected. 
Diagnostic  tests  give  information  about  gaps  in  student  knowledge  or  skill  that  serve  to 
guide  remediation.  Although  both  criterion-referenced  and  diagnostic  tests  are  needed  in 
instruction,  systematic  guidelines  for  their  development  were  incomplete  or  nonexistent 
in  existing  ISD  procedures  (Wetzel,  Ellis,  Wulfeck,  Sc  Montague,  1982).  The  IQI  classifica¬ 
tion  scheme  (see  Table  1)  provides  a  means  of  classifying  objectives  that  has  clear 
implications  for  test  development.  These  implications  have  been  specified  as  prescrip¬ 
tions  for  test  development,  and  guidelines  have  been  prepared  for  their  utilization  (Ellis  & 
Wulfeck,  1982;  Roid  Sc  Haladyna,  1982).  A  series  of  user  workshops  provided  formative 
evaluation  of  the  guidelines,  and  they  are  being  adopted  for  use  by  the  Naval  Education 
and  Training  Command. 

Limitations  and  Need  for  Additional  Development 


The  IQI  is  limited  in  what  it  attempts  to  accomplish.  There  are  some  important 
characteristics  of  instruction  to  which  the  IQI  does  not  attend.  One  concerns  adapting 
instructional  programs  to  individual  student  learning  styles,  experiential  background,  or 
preferences.  Glaser  (1S77)  discusses  the  theoretical  issues  but  provides  little  direct 
guidance  for  design. 

Another  problem  centers  around  the  analysis  of  content  structure.  Systematic 
approaches  to  instructional  design  emphasize  the  identification  of  requirements  in  the 
form  of  performance  objectives.  Although  this  was  an  important  advance  in  attempting 
to  develop  efficient  and  effective  training,  it  also  results  in  decontextualization.  An 
exhaustive,  detailed,  linear  progression  through  the  objectives  is  the  form  taken  by  much 
of  the  instruction  (Smith  Sc  Reigeluth,  1982).  Important  relationships  hetween  topics  are 
ignored,  and  rote  memorization  of  unorganized  facts  is  promoted.  A  student's  processing 
capacity  is  taxed  and  little  or  no  context  is  provided  for  retrieval  of  needed  information 
or  skill.  The  IQI  was  not  designed  to  handle  such  problems. 

Reigeluth,  Merrill,  Wilson,  and  Spiller  (1980),  in  developing  the  elaboration  theory  of 
instruction,  attempted  to  provide  prescriptive  recommendations  for  use  in  organizing  and 
structuring  performance  objectives.  Better  structure  and  sequence  should  improve 
learning  efficiency  and  retention.  It  is  important  to  begin  incorporating  these  ideas  into 
existing  evaluation  procedures. 

Need  for  Automation 


The  focus  on  the  consistency  and  adequacy  considerations  are  important  and  can  be 
used  to  correct  very  prevalent  errors  in  instruction.  Even  with  the  revisions  of  the 
methodology,  nowever,  judgments  about  adequacy  rely  considerably  on  the  knowledge  of 
instructional  practice  and  of  relevant  psychological  and  educational  research.  Since 


training  programs  for  this  knowledge  are  very  rare,  the  skills  are  likely  to  be  rare  among 
those  persons  who  are  responsible  for  instructional  development.  Although  techniques 
such  as  these  in  the  IQI  could  be  expanded  considerably  with  increasingly  more  detailed 
procedures  to  overcome  these  skill  deficiencies,  they  would  be  less  likely  to  be  used.  The 
key  appears  to  be  automation.  Computer-based  aids  can  reduce  the  apparent  complexity 
and  take  on  more  mundane  tasks,  allowing  designers  to  pay  more  attention  to  the  difficult 
tasks  of  analysis  and  design.  An  automated  version  of  the  IQI  could  speed  the  internal 
review  process  and  facilitate  corrections.  Automated  aids  for  instructional  development 
and  evaluation  are  currently  being  developed. 


RECOMMENDATIONS 

1.  It  is  recommended  that  instructional  developers  use  IQI  to  (a)  aid  in  the 
formative  evaluation  of  newly-developed  instruction,  (b)  review  existing  instruction  for 
consistency  and  adequacy,  and  (c)  provide  standards  for  contractor-developed  instruction. 

2.  It  is  recommended  that  IQI  procedures  be  automated  to  (a)  minimize  skill 
deficiencies  in  instructional  development  personnel,  (b)  allow  designers  to  pay  more 
attention  to  the  difficult  tasks  of  analysis  and  design,  and  (c)  speed  the  internal  review 
process  and  facilitate  revisions. 
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